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This  article  provides  perspectives  on  how  a  test  organization  can  organize  and  plan  for 
enterprise-wide  adoption  of  advances  in  emerging  technologies  and  techniques,  whether 
developed  in-house  or  acquired  from  external  sources.  This  article  enumerates  capabilities  that 
greatly  enhance  a  test  organizations  ability  to  support  the  impending  testing  demands  from  such 
GIG/SOA-based  projects  and  presents  an  overarching  strategic  plan  for  integrating  existing  test 
technologies,  identifying  enterprise-wide  technology  gaps,  and  coordinating  the  development 
and  acquisition  of  new  test  capabilities  to  greatly  accelerate  their  readiness  to  meet  impending 
net-centric  testing  challenges.  The  plan  discussed  in  this  article  includes  short-,  medium-,  and 
long-term  horizon  components  to  acquire  or  improve  current  test  capabUities  and  offers  a  layered 
architecture  that  provides  a  framework  for  capability  acquisition.  Test  organizations  can 
incentivize  their  contractors  to  exploit  the  composability,  reusability,  and  extensibility  of 
technical  attributes  of  SOA  to  support  the  development  of  the  layered  architecture.  The  authors 
conclude  that  the  design  of  the  test  organization  instrumentation  and  automation  on  top  of  the 
GIG/SOA  infrastructure  should  be  based  on  a  model-driven  software  approach,  systems- 
engineering  modeling,  and  simulation  principles  and  frameworks. 
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Given  Department  of  Defense  (DoD) 
mandates  for  transition  to  net-centric 
operation,  a  test  organization  must 
acquire  the  ability  to  perform  large- 
scale  and  fast-paced  developmental 
and  operational  testing  of  Global  Information  Grid/ 
Service  Oriented  Architecture  (GIG/SOA)-based  de¬ 
velopment  projects.  For  example,  the  Joint  Interoper¬ 
ability  Test  Command  has  the  responsibility  to  test  for 
GIG/SOA  compbance  for  such  projects  as  Net-Centric 
Enterprise  Services  and  Net-Enabled  Command  Ca¬ 


pability.  A  test  organization’s  abiUty  to  support  the 
impending  testing  demands  from  such  GIG/SOA- 
based  projects  can  be  greatly  enhanced  by  acquiring  net- 
centric  test  capabilities.  Although  most  test  organiza¬ 
tions  already  have  the  necessary  capabilities  to  some 
extent,  they  can  benefit  from  an  overarching  strategic 
plan  for  integrating  existing  test  technologies,  identify¬ 
ing  enterprise-wide  technology  gaps,  and  coordinating 
the  development  and  acquisition  of  new  test  capabilities 
to  greatly  accelerate  their  readiness  to  meet  impending 
net-centric  testing  challenges. 
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Net-centric  test  capabilities 

Several  specific  capabilities  that  a  test  organization 
must  address  to  effectively  conduct  developmental  and 
operational  tests  of  net-centric  systems  are  described 
below  (Buchheister  2005,  Carstairs  2005). 

Composability 

Composability  is  the  capability  to  seamlessly  com¬ 
pose  the  elements  of  the  desired  test  environment  by 
selecting  and  configuring  live  (e.g.,  human  players, 
military  systems)  and/or  virtual  (digital  representations 
of  live  components)  versions  of  all  test  environment 
components.  Test  organizations  can  take  advantage  of 
the  SOA  and  component  styles  that  offer  technical 
advantages  for  the  composition  of  test  instrumentation 
services  and  applications.  Contractors  should  be 
incentivized  to  exploit  the  SOA  constructs  to  huild 
plug-and-play  capabilities  while  meeting  current  and 
future  needs. 

Reusability  and  persistence 

The  test  infrastructure  persists  over  time  and 
includes  organized  repositories  to  support  the  reuse 
of  such  elements  as  simulation  models/digital  repre¬ 
sentations,  test  development  and  implementation 
processes,  and  test  experimentation  components  and 
tools  (intelligent  test  agents,  for  example).  This 
includes  the  capability  to  automatically  store,  catalog, 
and  retrieve  all  information  produced  by  any  node  on 
the  network  in  a  comprehensive,  standard  repository.  A 
critical  advantage  of  such  repositories  for  the  test 
organization  is  that  they  also  help  to  avoid  duplication 
of  efforts  by  the  test  organization’s  multiple  contrac¬ 
tors. 

Extensibility 

The  test  infrastructure  can  be  efficiently  extended 
through  the  use  of  common  architecture,  interfaces, 
processes,  and  tools.  Extensibility,  composability,  and 
reusability  are  mutually  supportive  attributes  of  model- 
driven  software  design  methodology  informed  by 
engineering  modeling  and  simulation  fundamentals. 
The  test  organization  must  incentivize  contractors  to 
adopt  such  methodologies  to  achieve  composability, 
reusability,  and  extensibility  attributes  in  its  develop¬ 
ments. 

Instrumented  trustworthy  measurement 

Instrumented  trustworthy  measurement  is  the  ability 
to  instrument  test  environments  in  a  manner  that  is 
principally  nonintrusive  and  highly  embedded,  which 
provides  real-time  measures  at  the  system  and  system- 
of-system  (SoS)  levels.  Measurement  is  consistent  and 
repeatable  across  experimental  replications,  providing 


reliable  and  trustworthy  data.  Specifically,  instrument¬ 
ed  trustworthy  measurement  includes  the  capability  to 

•  Reproduce  the  test  environment  and  play  back 
segments  of  the  test  event  in  a  manner  that 
facilitates  assessing  the  effects  of  modifying  the 
experimental  conditions  with  plug-and-play  re¬ 
placeable  test  components. 

•  Measure,  compare,  and  evaluate  experimentally 
specified  architectural  and  parametric  configura¬ 
tions  of  the  system  under  test. 

•  Collect  and  segregate  operational  data  (e.g., 
tactical  and  strategic  data  exchanged  between 
systems  under  test)  from  test  support  data  (e.g., 
instrumentation,  simulation,  analysis,  and  test 
control  data). 

•  Seamlessly  switch  between  real-time  and  after¬ 
test  analysis  of  collected  data. 

•  Perform  the  testing  of  net  ready  key  performance 
parameters  (NR-KPP)  and  compliance  to  the 
Net-Centric  Reference  Model  for  upcoming 
GIG/SOA  and  other  net-centric  developments. 

Visibility  and  controllability 

As  net-centric  systems  under  test  become  increas¬ 
ingly  complex,  the  ability  to  visualize  complex 
interactions  and  exert  control  over  such  interactions 
becomes  increasingly  vital  for  the  test  organization’s 
ability  to  provide  credible  test  results. 

Real-time  interactivity 

Real-time  interactivity  includes  visibility  into  events 
and  processes  through  a  display/representation  of  the 
test  environment  that  is  tailorable  and  provides 
accurate  situational  awareness  of  the  test  infrastructure 
and  the  tests  that  are  underway.  Currently,  many  test 
environments  focus  on  relatively  simple  interactions 
and  do  not  allow  for  highly  complex  many-on-many 
scenarios  in  which  test  environment  components 
(networks,  systems,  and  forces)  react  within  a  dynamic, 
closed-loop  environment. 

Features  of  advanced  test  organizations 

The  test  organization  should  strive  to  be  on  the 
cutting  edge  of  test  organization  capabilities,  including 

•  Agility.  Ability  to  automatically  and  adaptively 
monitor  and  manage  selective  functioning  of  the 
test  infrastructures,  test  scenarios,  networks,  and 
systems  and  services  under  test. 

•  Automation.  Ability  to  continually  enhance  the 
degree  of  automation  of  all  the  processes  involved 
in  defining,  implementing,  managing,  reusing, 
and  executing  test  events.  This  includes  auto¬ 
mated  self- organizing  recognition,  initialization. 
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and  control  of  plug-and-play  test  environment 
components. 

•  Scalability  and  Applicability  to  FuU  Life  Cycle. 
Ability  to  scale  the  test  infrastructure  in  terms  of 
size,  fidelity,  and  numbers  of  participants  to 
accommodate  the  domains  of  systems  engineering, 
development,  development  testing,  operational 
testing,  interoperability  certification  testing,  and 
net-readiness  and  information  assurance  testing. 

•  GIG/SOA  Integrated  Robust  Computer  and 
Communication  Infrastructure.  Ability  to  provide 
high-performance  computational  support  wherev¬ 
er  needed  in  the  configuration  and  execution  of  the 
test  environment  and  the  analysis  of  test  data  (in 
real  time  and  after  test).  As  the  SoS  and 
collaborations  brought  in  by  customers  for  testing 
become  increasingly  complex,  the  test  organization 
will  require  increasingly  powerful  computing 
resources  to  manage  aU  aspects  of  testing.  The 
test  organization  will  also  require  the  ability  to 
provide  reliable,  cost-effective,  flexible,  and  GIG- 
enabled  communication  to  all  nodes. 

(Note:  Most  of  these  requirements  are  not  achievable 
with  current  manually  based  data  collection  and 
testing.  Instrumentation  and  automation  based  on 
model-driven  and  systems-engineering  modeling  and 
simulation  principles  and  frameworks  are  needed  to 
meet  these  requirements.) 

Proposed  Acquisition  Strategy 

Acquiring  all  the  assets  needed  for  the  above 
capabilities  would  significantly  upgrade  the  test 
organization’s  capability  for  net-centric  testing,  but 
they  will  vary  in  degree  of  maturity.  Some  may  be  ready 
for  implementation  or  purchase  in  the  near  term,  and 
others  may  require  significant  investment  in  research 
and  development.  To  help  manage  the  acquisition  of 
such  assets,  we  propose  an  acquisition  strategy  having 
three  levels  corresponding  to  long-,  medium-,  and 
short-term  planning  horizons:  (a)  overall  plan  for  test 
infrastructure  evolution,  (b)  test  infrastructure  devel¬ 
opment  to  address  test  technology  shortfalls,  and  (c) 
planning  for  individual  test  venues  and  events  (Fig¬ 
ure  1).  The  underlying  objective  of  the  proposed 
strategy  is  to  foster  re-use  of  existing  assets  so  as  to 
maximize  the  cost-effectiveness  of  acquisition.  The 
goal  should  be  to  set  up  a  process  for  re-use,  so  that 
new  capabilities  are  needed  only  when  existing  ones 
cannot  be  reasonably  applied  to  the  new  situation. 

Planning  levels 

Long-term  pianning 

With  respect  to  long-term  planning,  the  objective  is 
to  look  out  past  the  horizon  of  imminent  test  events  and 


Long  term — Overall  evolution  of  net-centric 
T&E  infrastructure. 


Medium  term — Test  infrastructure 
development  plan  to  address  test  technology 
shortfalls. 


Short  term — Planning  for  individual  test 
venues  and  events. 


Figure  1.  Net-centric  testing  pianning  levels 

current  infrastructure  improvement  projects  to  identify 
emerging  technologies  and  emerging  system  objectives 
and  to  lay  out  the  broad  approach  to  development  of  the 
test  and  evaluation  infrastructure.  As  Figure  2  illus¬ 
trates,  we  suggest  a  planning  approach  to  test  individual 
customer  projects  and  test  events  as  part  of  the  longer 
life  cycle  of  the  test  infrastructure  evolution.  Key 
activities  in  the  long-term  strategic  plan  are  as  follows. 

As  new  systems  are  defined  and  developed  by  a 
customer  that  will  be  subject  to  the  test  organization 
certification,  the  test  organization  must  derive  a 
coherent  family  of  test  objectives  from  the  stated  or 
to-be-developed  system  under  test  requirements  and 
behavior  specifications.  Test  events,  venues,  and 
infrastructure  evolution  must  be  synchronized  with 
the  customer  system  development  schedule. 

The  high-level  characteristics  of  the  test  develop¬ 
ment  methodology  and  of  the  infrastructure  to  be  used 
must  be  determined  to  meet  the  perceived  complexity, 
volume,  variety,  and  velocity  of  test  challenges — ^with 
the  objectives  of  furthering  re-use  of  test  resources  and 
fostering  cumulative  knowledge  management.  This 
includes,  among  other  things,  establishing  require¬ 
ments  for  infrastructure  development  tools,  such  as 
formalizing  and  designing  test  models. 

This  long-term  planning  process  passes  technical 
shortfalls  and  their  temporal  attributes  (e.g.,  “needed 


Figure  2.  Long-term  cycle  of  test  activities 
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Figure  3.  System-specific  and  individual  event  planning  cycle 


immediately,”  “needs  can  be  foreseen  for  tests  sched¬ 
uled  in  the  near  future,”  or  “is  not  critical  now”)  on  to 
medium-term  planning. 

Medium-  and  short-term  planning 

The  planning  for  individual  test  venues  and  events 
consists  of  a  cycle  of  activities  that  work  within  the 
structure  established  by  the  high-level  planning.  As 
Figure  3  illustrates,  this  cycle  consists  of  the  following 
basic  elements: 

Establish  objectives.  The  test  objectives  must  provide 
an  overview  of  the  high-level  system-specific  test 
objectives  and  identify  basic  technical  and  operational 
evaluations  that  are  needed  to  support  future  decision 
events.  The  objectives  must 

•  Be  tied  to  the  system  acquisition  strategy. 

•  Establish  the  basis  for  a  test  and  evaluation 
schedule  in  terms  of  test  capabilities  that  will  be 
available  after  each  iteration  of  the  test  and 
evaluation  process — this  should  include  both 
anticipated  costs  and  timelines.  It  is  vital  that 
the  test  organization  and  the  customer  agree  to 
an  integrated  budget  and  timeline  for  each  test 
objective. 

•  Be  coordinated  with  the  customer’s  strategy  for 
system  development  and  demonstration. 

•  Identify  major  strategic  risks  to  achieving  the 
identified  test  capabilities  and  lay  out  the 
activities  necessary  to  mitigate  the  risks. 

•  Identify  challenges,  such  as  from  complexity  and 
need  for  testing  that  cannot  be  accomplished 


manually  in  sufficient  volume,  which  must  be 
overcome  to  effectively  assess  SoS  and  systems  to 
contribute  to  their  improvement.  Update  plans  to 
meet  these  challenges. 

Identify  relevant  test  environment  requirements. 
Once  the  test  objectives  are  set,  identify  and  evaluate 
specific  test-support  capabilities  with  respect  to  how 
they  contribute  to  satisfying  the  test  objectives.  At  this 
stage,  a  test  environment  description  is  constructed, 
which  is  tailored  to  the  test  objectives;  relevant 
capabilities  of  the  system  under  test  are  identified,  and 
testable  metrics  are  developed  for  those  capabilities. 

Reuse/build  scenarios  and  mission  threads  to 
exercise  given  system  under  test  requirements. 
The  list  of  requirements  for  the  system  under  test  is 
linked  to  the  underlying  operational  concepts  and 
capabilities.  With  this  list  in  hand,  it  is  vital  to  develop 
specific  mission  threads  that  exercise  these  capabilities 
in  a  way  that  is  relevant  to  the  test  objectives  and 
anticipated  operational  environment. 

Identify  atomic  functional  units,  decompose  such 
functions  into  atomic  behaviors,  and  implement 
test  behaviors.  The  preceding  three  activities  set  the 
stage  for  technical  development  of  the  test  environment. 
The  technical  development  phase  includes  (a)  identify¬ 
ing  the  atomic  functional  units  of  the  system  under  test 
that  comprise  the  identified  capabilities,  (b)  decompos¬ 
ing  these  functional  units  into  atomic  testable  behaviors, 
(c)  combining  these  test  behaviors  as  test  models  that 
can  be  compared  with,  and  operated  against,  the  system 
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under  test  in  the  test  environment.  At  this  point,  specific 
system  under  test  components  and/or  subsystems  are 
identified  as  being  relevant  to  specific  system  capabilities 
in  the  context  of  identified  mission  threads,  and  the  test 
machinery  needed  to  stimulate  and  observe  these 
components  is  ready  to  be  put  into  place. 

Build  and/or  reuse  test  bed  software  and  hardware 
for  executing  test  models;  design  and  execute 
test  events.  Test  events  are  planned  to  apply  specific 
test  bed  items  to  the  system  under  test.  The  test  plan 
includes  a  test  environment  configuration  for  the  test 
events,  identifies  the  source  of  test  data  (e.g.,  live  data, 
recorded  system  traces,  simulations),  and  sets  specific 
pass/fail  criteria  for  the  event.  Acquire,  build,  and/or 
improve  infrastructure  development  tools,  such  as  tools 
for  formalizing  and  designing  test  models. 

This  cycle  of  test  activities  defines  an  iterative  process 
that  allows  for  the  evolution  of  each  test  phase  as  the 
system  under  test  moves  through  its  life  cycle  (Figure  3). 
Throughout  the  cycle  of  test  activities,  there  must  be  an 
emphasis  on  the  reuse  of  proven,  reliable,  and  efficient 
infrastructure  elements  and  artifacts  that  were  acquired 
as  a  result  of  earher  test  projects.  Efforts  first  capitalize 
on  reusing  existing  software  and  hardware  for  executing 
test  models.  Of  course,  the  requirements  of  each  new 
project  may  exceed  the  capabilities  of  the  current 
infrastructure  and  artifacts,  in  which  case  we  seize 
opportunities  to  enhance  the  infrastructure.  Thus,  each 
specific  system  under  test  feeds  back  lessons  learned  and 
contributes  to  long-term  capabilities  and  knowledge. 
This  feedback  loop  is  illustrated  in  Figure  2. 

Proposed  layered  architecture 

To  support  the  acquisition  of  net-centric  testing 
capability  with  the  time  horizons  just  discussed,  we  offer 
a  layered  architecture  that  provides  a  framework  for  such 
capability  acquisition.  We  propose  that  the  test 
organization  develop  an  overall  architecture  for  net- 
centric  instrumentation  as  illustrated  in  Figure  4.  The 
architecture  is  based  on  that  presented  in  Sarjoughian, 
Xiegler,  and  HaU  2001  and  refers  to  background  in 
literature  on  modeling  and  simulation  (Zeigler,  Fulton, 
Hammonds,  and  Nutaro  2005;  Zeigler,  Kim,  Praehofer 
2000;  Zeigler  and  Hammonds  2007;  Traore  and  Muxy 
2004);  Systems  of  Systems  (Sage  2007;  Wymore  1992; 
Wymore  1967;  Morganwalp  and  Sage  2004);  model- 
driven  software  development  (Dimario  2007;  Dimario 
2006;  Object  Modeling  Group  2007;  Jacobs  2004; 
Wagenhals,  Haider,  and  Levis  2002;  Wegmann  2002); 
and  integrated  simulation-based  development  and 
testing  (Mak,  Mittal,  and  Hwang  [in  press];  Mittal 
2006;  Mittal,  Mak,  and  Nutaro  2006;  Mittal  2007; 
Mittal,  Sahin,  and  Jamshidi  [in  press]). 


Figure  4.  Architecture  for  net-centric  test  instrumentation 


Network  layer 

The  network  layer  contains  the  actual  computers 
(including  workstations  and  high  performance  systems) 
and  the  connecting  networks  (both  local  area  network 
and  wide  area  network,  their  hardware  and  software). 

Execution  layer 

The  execution  layer  is  the  software  that  executes  the 
models  in  simulation  time  and/or  real  time  to  generate 
their  behavior.  Included  in  this  layer  are  the  protocols 
that  provide  the  basis  for  distributed  simulation  (such 
as  those  that  are  standardized  in  the  high  level 
architecture).  Also  included  are  database  management 
systems  and  software  for  controlling  simulation 
executions  and  for  displaying  test  results  and  animated 
visuals  of  the  behaviors  generated. 

Modeling  layer 

The  modeling  layer  supports  the  development  of 
simulation  models  and  other  digital  representations  for 
net-centric  testing  in  formalisms  that  are  independent 
of  execution  layer  implementations.  At  this  layer,  the 
test  organization  would  compose  services  and  applica¬ 
tions.  Also  in  this  layer  is  support  for  the  quality 
control  of  model  acquisition,  especially  the  key 
processes  of  verification  and  validation  of  models, 
simulators,  and  test  tools. 

Experimental  frame  layer 

The  experimental  frame  layer  employs  the  artifacts 
and  services  of  the  modeling  layer  to  develop  test 
components,  such  as  generators,  acceptors,  and  trans¬ 
ducers  and  their  compositions,  to  provide  test  instru¬ 
mentation  services.  Included  are  the  observers  and 
agents  that  run  in  the  execution  layer,  and  that 
interface  with  the  systems  and  services  under  test  to 
connect  them  to  the  experimental  frame  components. 
Also  included  are  means  to  capture  relevant  measures 
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of  performance  and  effectiveness  and  instrument  them 
as  experimental  frame  compositions  employing  mod¬ 
eling  layer  and  execution  layer  services.  These  measures 
are  critical  to  the  testing  of  NR-KPPs  that  the  test 
organization  must  be  able  to  accomplish. 

Design  and  test  development  layer 

The  design  and  test  development  layer  supports  the 
ingestion  and  analysis  of  model-based  system  specifica¬ 
tion  documents,  such  as  in  the  DoD  Architecture 
Framework,  where  the  design  is  based  on  specifying 
desired  behaviors  through  models  and  implementing 
these  behaviors  through  interconnection  of  system 
components.  In  the  modeling  layer,  results  of  this 
analysis  of  system  behavior  requirements  will  be  used 
with  automated  generation  of  test  models,  which  when 
deployed  in  the  execution  layer  as  automated  test  cases 
will  interact  with  systems  and  services  under  test.  The 
design  and  test  development  layer  also  includes 
maintenance  and  configuration  support  for  large 
families  of  alternative  test  architectures,  whether  in  the 
form  of  spaces  set  up  by  parameters  or  more  powerful 
means  of  specifying  alternative  model  structures  such  as 
provided  by  the  System  Entity  Structure  (SES) 
methodology.  Artificial  intelligence  and  simulated 
natural  intelligence  (evolutionary  programming)  may 
be  brought  in  to  help  deal  with  combinatorial  explosions 
occasioned  by  analysis  for  test  development. 

Collaboration  and  customer  interaction  layer 

The  collaboration  and  customer  interaction  layer 
enables  people  and/or  intelligent  agents  to  manage  and 
control  the  infrastructure  capabilities  supplied  by 
underlying  layers.  This  includes  interactions  with  the 
customer  in  which  test  results  are  conveyed  and 
explained  if  needed. 

Note  that  these  layers  describe  functionalities  that 
can  be  partially  supplied  by  proven  and  reliable  legacy 
tools  in  the  test  organization’s  inventory  from  earlier 
developments.  However,  the  primary  objective  of  such 
architecture  is  to  facilitate  carrying  out  the  multi¬ 
horizon  planning  approach  discussed  earlier.  As 
customer  projects  arrive,  their  testing  requirements 
can  be  referenced  to  the  elements  within  the  layered 
architecture — the  detailed  test  assets  at  the  various 
levels  are  called  out.  Missing  assets  can  be  the  cues  to 
start  an  acquisition  process  to  fill  the  gap.  Figure  6 
illustrates  the  application  of  the  layered  architecture  to 
sensor  simulation  infrastructure  acquisition. 

Artifacts,  such  as  models  and  test  and  evaluation  are 
results  of  processes  (systems)  that  must  not  only  have 
hardware  and  software  support  but  must  be  done  by 
competent  people  using  competent  methods  in  an 
environment  that  fosters  each  process.  Indeed,  to  be 


Figure  5.  The  layered  architecture  viewed  from  the 
DOTMLPF  perspective 


effective,  there  must  be  collaboration  among  layers  and 
continuity  of  people,  methods,  software  and  hardware, 
good  input  and  materials,  and  a  supportive  environment 
(e.g.,  from  management  and  external  networks).  This 
collaboration  is  illustrated  in  Figure  5,  employing  the 
basic  categories  of  People,  Policy  and  Methods, 
Hardware  and  Software,  Input  Data  and  Materials, 
and  Environment;  expressing  the  areas  DoD  often  refers 
to  as  DOTMLPF — doctrine,  organization,  training, 
materiel,  leadership,  personnel,  facilities.  To  better 
communicate  the  main  collaboration  path,  connections 
for  exception  handling  and  additional  feedback  have  not 
been  included  in  Figure  5.  We  recognize  that  a  real- 
world  portrayal  of  the  collaboration  would  include 
numerous  iterations,  feedback,  and  exception  handling. 

Table  1  suggests  how  some  of  the  identified  layers 
can  be  further  elaborated  in  terms  of  representative 
needs  that  must  be  met  in  the  basic  categories  that  are 
most  pertinent  to  each  layer. 

We  note  that  the  table  makes  clear  that  besides  the 
acquisition  and  application  of  test  infrastructure 
elements,  the  Joint  Interoperability  Test  Command 
(JITC)  must  plan  for  acquiring  the  right  personnel  and 
instituting  the  right  organization.  Specifically,  JITC 
must  develop  a  culture  that  wiU  facilitate  the 
interactions  among  personnel  that  are  critical  for  the 
enterprise  to  be  effective. 

Mapping  shortfalls  to  architectural  layers 

The  proposed  layered  architecture  will  provide  a 
framework  for  focusing  the  planning  and  acquisition  of 
the  test  infrastructure  capability.  With  the  Xs  in  the 
cells  of  Table  2  we  offer  a  mapping  to  the  shortfall 
areas  that  we  think  are  best  addressed  in  each  layer. 
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Table  1.  Illustrating  the  layered  architecture  in  relation  to  doctrine,  organization,  training,  materiel,  leadership,  personnel,  and 

facilities  (DOTMLPF) 

Layer 

People,  Policy,  and  Methods 

Hardware  and  Software  Input  Data  and  Materials 

Environment 

Experimental 

Experimental  Frame  Developers 

(1)  Access  to  relevant 

(1)  V6cV  experimental 

(1)  Development,  testing, 

Frame  Layer 

(1)  are  qualified,  (2)  have 

models  and  software 

frame  artifacts  and 

and  V8cV  are  managed 

methodologies  that  are 

to  gather  required 

test  components  from 

to  plan.  (2)  Proper  SW 

appropriate  and  effective, 

measures  (MOEs, 

the  Modeling  Layer. 

CM  environment  and 

(3)  have  shared  awareness  of 

MOPs),  generate 

(2)  V&V ed  data  for 

practice. 

development  plans,  design 

required  stimuli  and 

DT,  &V,  VScT.  (3) 

decisions,  and  progress,  and 

loads,  and  control. 

Good  requirements 

(4)  have  good  access  to  model 

(2)  Model  development 

and/or  standards. 

developers  and  to  test  development 

tools  and  software 

(4)  V6cVed  means  to 

personnel  who  are  prepared  to 

integrated  design 

capture  relevant 

clarify  requirements  and  standards 

environments  are 

measures. 

governing  the  systems  under  test. 

adequate.  (3)  Access 
to  JITC  network  and 
to  test  workstations. 

Design  and  Test 

Design  and  Test  Developers 

Adequate  tools  to 

(1)  Adequate  system 

(1)  Unplanned  requirement 

Development 

(1)  are  qualified,  (2)  have 

capture  and 

specification 

additions  are  avoided. 

Layer 

methodologies  that  are 

characterize  systems 

documents  and 

(2)  Proper  CM 

appropriate  and  effective, 

under  test  behaviors  and 

DoDAF  documents. 

environment  and 

(3)  have  shared  awareness  with 

interfaces. 

(2)  Behavior 

practice. 

the  JITC  team,  and  (4)  have 

requirements  and/or 

good  access  to  personnel  who  are 

standards  are  sufficiently 

prepared  to  clarify  requirements 

well-specified.  This 

and  standards  governing  the  systems 

applies  particularly  to 

under  test. 

GIG/SOA-based 

developments  (e.g., 

NCES,  NECC). 

The  test  organization  should  employ  this  architecture 
as  the  basis  for  its  net-centric  instrumentation  plan. 

Strategies  for  net-centric 
instrumentation  pianning 

With  the  layered  architecture  as  basis,  the  test 
organization  can  develop  specific  strategies  that  take 


into  account  long-,  medium-,  and  short-term  consid¬ 
erations  for  orderly  acquisition  of  effective  and  reusable 
infrastructure.  One  alternative  is  to  continue  to  rely  on 
legacy  tools  while  employing  the  architecture  to  plan  for 
new  tool  acquisitions  as  the  opportunities  present 
themselves.  Another  alternative  is  to  invest  immediately 
in  high  priority  tool  developments  that  are  compliant  to 


Table  2.  Illustrating  the  mapping  of  shortfalls  in  architectural  layers 


Layers 


Experimental 

Design  and  text 

Collaboration  and 

Network 

Execution 

Modeling 

frame 

development 

customer  interaction 

Composability 

X 

X 

X 

Reuseability  and  persistence 

X 

X 

X 

X 

Extensibility 

X 

X 

X 

Instrumented  trustworthy 

X 

measurement 

Visibility  and  controllability 

X 

X 

X 

Real-time  interactivity 

Agility 

X 

X 

X 

X 

Automation 

X 

X 

X 

X 

X 

X 

Scalability  and  applicability  to  full 

X 

X 

X 

X 

life  cycle 

GIG/SOA  integrated  robust 

X 

X 

X 

X 

X 

X 

computer  and  communication 
infrastructure 


GIG/SOA,  Global  Information  Grid/Service  Oriented  Architecture. 
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Multi  system 
interoperability 


Reusable  infrastructure 

Establish 

Identify  relevant 

for  JITS  ,  AW  AGS, 

objectives 

system-under-test 

lABM  testing 

requirements 

Design  and  execute  test  events 
that  implement  mission  threads 
using  the  current  test  environ¬ 
ment  instantiation 


Reuse  existing  test  bed  soft¬ 
ware/hardware  for  executing 
test  models — augment  when 
necessary 


Inject  radar  data  into 
sensors  and  cause  them 
to  attempt  correlation 


E.g.,  track  processing, 
management,  and  correla¬ 
tion  transactions 


Migrate  to  GIG — compatible  net¬ 
working  protocols  to  support 
composability  &  reusability 


Employ  ATC-Gen  to 
control  sensor  injected 
data  and  observe 
correlation  responses 


Implement  test 

Decompose 

behaviors  as  test 

functions  into 

models  that  can 

low-level 

be  federated  with 

behaviors  that 

the  system  under 

implement  the 

test 

functions 

Reuse/build  scenarios 
and  mission  threads  to 
exercise  requirements 


Identify  low-level 
functions  from 
capabilities  list  and 
mission  threads 


Eorrnalize  Link- 16  rules 
that  are  involved  in 
identified  transactions 


Figure  6.  Illustrating  event  planning  cycie  for  sensor  simulation  acquisition 


such  architecture  and  that  implement  nonexistent 
capabilities  such  as  planning  or  automated  testing  and 
may  not  replace  legacy  tools  in  the  near  term. 

Illustrative  application  to  sensor  simulation 
infrastructure  acquisition 

Figure  6  sketches  how  the  planning  cycle  of  Figure  2 
might  apply  to  the  acquisition  of  sensor  simulation  for 
net-centric  testing.  The  perspectives  offered  by  multi¬ 
horizon  planning  and  layered  test  infrastructure 
architecture  are  intended  to  facilitate  developing  and 
evaluating  acquisition  strategies.  By  themselves,  they 
do  not  decide  the  choices  to  make. 

Summary  and  recommendations 

A  test  organization  needs  an  instrumentation 
development  and  maintenance  system  that  can  be 


considered  an  open  subsystem  of  an  open  system — the 
test  organization,  test  evaluation,  and  certification 
system,  which  produces  results  as  shown  on  the  left 
side  of  Figure  7.  Shown  on  the  left  are  the  resources 
and  funds  leaving  the  system,  and  on  the  right  are  the 
funds  and  resources  coming  in.  In  addition,  entering  at 
the  right  is  a  seemingly  high  volume  of  a  broad  variety 
of  not  always  clear  or  fixed  system-under-design 
requirements,  protocols,  waveforms,  standards,  and 
mandated  architectural  styles  (e.g.,  net-centric  refer¬ 
ence  model  and  SOA).  As  shown  at  the  bottom  right, 
the  test  organization  must  encourage  scientific  research 
and  technology  development  projects  of  the  govern¬ 
ment,  academia,  and  industry  to  develop  methods  and 
technologies  needed  to  fiU  test  capability  gaps. 

The  specific  inclusion  of  infrastructure  development 
as  an  integral  part  of  the  top-down  approach  fosters 
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Test  and/or 

Certification 

Reports 

◄ - 

Self-created 
methodology 
and  technology 

< - 

Resources* 
and  funds 
leaving  the 
system 

A - 


Organizational  Test,  Evaluation, 
and  Certification  System 


Customer  and  DoD 
requirements, 
objectives,  access  to 
SUTs,  and  funds 


♦ 


Instrumentation  Development 
and  Maintenance  System 


Test 

network 

1 


Test 

network 

n 


Emerging  proto¬ 
cols,  standards, 
waveforms, 
principles, 
methodologies,  and 
arcliitectural  styles 


^ - 

Resources* 

A - 


New  methods  and 
technologies  from 
T&E  sponsored 
S&T  projects 


*  Resources  =  Peopie,  hardware  &  software,  RF  and  IP  network  services,  and  materials. 

Figure  7.  Instrumentation  development  and  maintenance  subsystem  of  the  test  organization  test  and  evaluation  and 
certification  system 


significant  reuse  of  test  resources  and  cumulative 
knowledge  management  of  the  products  of  testing. 
We  recommend  that  in  addition  to  basic  test 
development,  each  iteration  of  the  individual  test 
event/venue  planning  cycle  should  a/so  target  a  small, 
well-defined,  and  incremental  enhancement  of  the  test 
environment  functionality  that  we  implement  as  compo¬ 
nents  of  the  overall  test  infrastructure.  Iterations  should 
refine  and/or  enhance  test  objectives  and  develop  and/ 
or  modify  the  test  bed  technology  as  needed;  and  test 
events  should  realize  these  test  objectives  using  the 
available  test  bed  capabilities.  In  addition  to  supporting 
the  planned  test  objectives,  each  iteration  should  to  the 
extent  possible  include  a  test  event  that  specifically 
demonstrates  the  new  test  environment  functionality. 

Testing  in  this  paradigm  is  objective  driven  rather 
than  event  driven  (i.e.,  test  events  must  be  traceable 
back  to  established  test  objectives).  In  most  cases, 
major  shortfalls  of  test  technology  should  be  identified 
early,  either  during  the  refmement/expansion  of  test 
objectives,  or  in  the  early  phases  of  test  event  planning. 
Interim  technology  solutions  to  reduce  shortfalls  that 
are  identified  late  in  test  event  planning  or  even  later 
during  test  event  execution  should  be  considered 
tentative  pending  review  in  the  next  iteration  of  the 
test  bed  development.  These  interim  solutions  should 
be  the  exception  and  not  the  rule. 

We  recognize  that  infrastructure  development  re¬ 
quires  competent  people  using  competent  methods  in  an 


environment  that  fosters  the  development  of  each 
process  and  artifact.  In  this  regard,  we  recommend 
including  in  the  test  organization  team  a  test-infra¬ 
structure  development  component  that  supports  testing 
for  each  customer  project  and  its  test  events.  The 
responsibilities  of  this  infrastructure  team  would  be  to 

•  Identify  existing,  reusable  testing  tools  and 
requirements  that  are  common  across  test  activ¬ 
ities  for  use  and  for  potential  adaptation  or 
conversion  to  a  reusable  component. 

•  Build  and  maintain  reusable  technical  compo¬ 
nents  of  a  common  test  infrastructure. 

•  Promote  test  asset  reuse  where  appropriate. 

•  Advise  test  event  planning  and  execution  when 
the  events  rely  on  pieces  of  the  common  test 
infrastructure. 

•  Retain  and  disseminate  lessons  learned  from  a 
test  event. 

In  addition  to  the  net-centric  test  infrastructure 
components  involved  in  specific  customer  projects,  the 
test  organization  should  stand  up  a  global  test  infrastruc¬ 
ture  development  team  to  operate  within  the  larger 
framework  of  its  enterprise  level  plans  for  coordinating 
instrumentation,  automation,  and  architecture  support 
across  all  the  test  organization  portfolios.  This  team  would 

•  Coordinate  efforts  for  customer-specific  devel¬ 
opments  with  the  test  organization’s  enterprise 
level  net- centric  test  infrastructure  development 
and  identify  overlapping  concerns  and/or  testing 


29(1)  •  March  2008  59 


Bridges,  Zeigler,  Nutaro,  Hall,  et  al. 


tools.  Customer-specific  testing  requirements  can 
be  referenced  to  the  elements  within  the  layered 
architecture,  calling  out  detailed  test  assets  at  the 
various  levels.  Missing  assets  can  be  the  cues  to 
start  acquisitions. 

•  Provide  proactive  technical  solutions  to  identified 
customer-specific  test  requirements.  These  solu¬ 
tions  wiU  be  incorporated  into  test  events  that 
wiU  be  planned  in  detail  later  on  in  the  test  and 
evaluation  process. 

•  Seek  out  and  recommend  best  practices  and  cultural 
innovations  that  wUl  faciUtate  effective  coordination 
of  the  personnel  working  at  the  various  architectural 
layers  as  customer  projects  arrive. 

•  Participate  actively  in  teams  responsible  for  test 

planning  and  developing  test  tools  for  specific 
events.  Successful  reuse  requires  positive  involve¬ 
ment  at  aU  levels  of  the  organization.  Conse¬ 
quently,  persons  responsible  for  long-term  infra¬ 
structure  development  must  be  constructively  and 
actively  engaged  with  the  elements  of  the 
organization  that  they  support.  □ 
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